Finding structure during incremental speech comprehension.

A core aspect of human speech comprehension is the ability to incrementally integrate consecutive words into a structured and coherent interpretation, aligning with the speaker's intended meaning. This rapid process is subject to multidimensional probabilistic constraints, including both linguistic knowledge and non-linguistic information within specific contexts, and it is their interpretative coherence that drives successful comprehension. To study the neural substrates of this process, we extract word-by-word measures of sentential structure from BERT, a deep language model, which effectively approximates the coherent outcomes of the dynamic interplay among various types of constraints. Using representational similarity analysis, we tested BERT parse depths and relevant corpus-based measures against the spatiotemporally resolved brain activity recorded by electro-/magnetoencephalography when participants were listening to the same sentences. Our results provide a detailed picture of the neurobiological processes involved in the incremental construction of structured interpretations. These findings show when and where coherent interpretations emerge through the evaluation and integration of multifaceted constraints in the brain, which engages bilateral brain regions extending beyond the classical fronto-temporal language system. Furthermore, this study provides empirical evidence supporting the use of artificial neural networks as computational models for revealing the neural dynamics underpinning complex cognitive processes in the brain.

An artificial intelligence-based pipeline for automated detection and localisation of epileptic sources from magnetoencephalography.

Objective.Magnetoencephalography (MEG) is a powerful non-invasive diagnostic modality for presurgical epilepsy evaluation. However, the clinical utility of MEG mapping for localising epileptic foci is limited by its low efficiency, high labour requirements, and considerable interoperator variability. To address these obstacles, we proposed a novel artificial intelligence-based automated magnetic source imaging (AMSI) pipeline for automated detection and localisation of epileptic sources from MEG data.Approach.To expedite the analysis of clinical MEG data from patients with epilepsy and reduce human bias, we developed an autolabelling method, a deep-learning model based on convolutional neural networks and a hierarchical clustering method based on a perceptual hash algorithm, to enable the coregistration of MEG and magnetic resonance imaging, the detection and clustering of epileptic activity, and the localisation of epileptic sources in a highly automated manner. We tested the capability of the AMSI pipeline by assessing MEG data from 48 epilepsy patients.Main results.The AMSI pipeline was able to rapidly detect interictal epileptiform discharges with 93.31% ± 3.87% precision based on a 35-patient dataset (with sevenfold patientwise cross-validation) and robustly rendered accurate localisation of epileptic activity with a lobar concordance of 87.18% against interictal and ictal stereo-electroencephalography findings in a 13-patient dataset. We also showed that the AMSI pipeline accomplishes the necessary processes and delivers objective results within a much shorter time frame (∼12 min) than traditional manual processes (∼4 h).Significance.The AMSI pipeline promises to facilitate increased utilisation of MEG data in the clinical analysis of patients with epilepsy.

Increasing diversity in connectomics with the Chinese Human Connectome Project.

Cultural differences and biological diversity play important roles in shaping human brain structure and function. To date, most large-scale multimodal neuroimaging datasets have been obtained primarily from people living in Western countries, omitting the crucial contrast with populations living in other regions. The Chinese Human Connectome Project (CHCP) aims to address these resource and knowledge gaps by acquiring imaging, genetic and behavioral data from a large sample of participants living in an Eastern culture. The CHCP collected multimodal neuroimaging data from healthy Chinese adults using a protocol comparable to that of the Human Connectome Project. Comparisons between the CHCP and Human Connectome Project revealed both commonalities and distinctions in brain structure, function and connectivity. The corresponding large-scale brain parcellations were highly reproducible across the two datasets, with the language processing task showing the largest differences. The CHCP dataset is publicly available in an effort to facilitate transcultural and cross-ethnic brain-mind studies.

Multimodal neuroimaging with optically pumped magnetometers: A simultaneous MEG-EEG-fNIRS acquisition system.

Multimodal neuroimaging plays an important role in neuroscience research. Integrated noninvasive neuroimaging modalities, such as magnetoencephalography (MEG), electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), allow neural activity and related physiological processes in the brain to be precisely and comprehensively depicted, providing an effective and advanced platform to study brain function. Noncryogenic optically pumped magnetometer (OPM) MEG has high signal power due to its on-scalp sensor layout and enables more flexible configurations than traditional commercial superconducting MEG. Here, we integrate OPM-MEG with EEG and fNIRS to develop a multimodal neuroimaging system that can simultaneously measure brain electrophysiology and hemodynamics. We conducted a series of experiments to demonstrate the feasibility and robustness of our MEG-EEG-fNIRS acquisition system. The complementary neural and physiological signals simultaneously collected by our multimodal imaging system provide opportunities for a wide range of potential applications in neurovascular coupling, wearable neuroimaging, hyperscanning and brain-computer interfaces.

A Heteromodal Word-Meaning Binding Site in the Visual Word Form Area under Top-Down Frontoparietal Control.

The integral capacity of human language together with semantic memory drives the linkage of words and their meaning, which theoretically is subject to cognitive control. However, it remains unknown whether, across different language modalities and input/output formats, there is a shared system in the human brain for word-meaning binding and how this system interacts with cognitive control. Here, we conducted a functional magnetic resonance imaging experiment based on a large cohort of subjects (50 females, 50 males) to comprehensively measure the brain responses evoked by semantic processing in spoken and written word comprehension and production tasks (listening, speaking, reading, and writing). We found that heteromodal word input and output tasks involved distributed brain regions within a frontal-parietal-temporal network and focally coactivated the anterior lateral visual word form area (VWFA), which is located in the basal occipitotemporal area. Directed connectivity analysis revealed that the VWFA was invariably under significant top-down modulation of the frontoparietal control network and interacts with regions related to attention and semantic representation. This study reveals that the VWFA is a key site subserving general semantic processes linking words and meaning, challenging the predominant emphasis on this area's specific role in reading or more general visual processes. Our findings also suggest that the dynamics between semantic memory and cognitive control mechanisms during word processing are largely independent of the modalities of input or output.SIGNIFICANCE STATEMENT Binding words and their meaning into a coherent whole during retrieval requires accessing semantic memory and cognitive control, allowing our thoughts to be expressed and comprehended through mind-external tokens in multiple modalities, such as written or spoken forms. However, it is still unknown whether multimodal language comprehension and production share a common word-meaning binding system in human brains and how this system is connected to a cognitive control mechanism. By systematically measuring brain activity evoked by spoken and written verbal input and output tasks tagging word-meaning binding processes, we demonstrate a general word-meaning binding site within the visual word form area (VWFA) and how this site is modulated by the frontal-parietal control network.

Decoding the Real-Time Neurobiological Properties of Incremental Semantic Interpretation.

Communication through spoken language is a central human capacity, involving a wide range of complex computations that incrementally interpret each word into meaningful sentences. However, surprisingly little is known about the spatiotemporal properties of the complex neurobiological systems that support these dynamic predictive and integrative computations. Here, we focus on prediction, a core incremental processing operation guiding the interpretation of each upcoming word with respect to its preceding context. To investigate the neurobiological basis of how semantic constraints change and evolve as each word in a sentence accumulates over time, in a spoken sentence comprehension study, we analyzed the multivariate patterns of neural activity recorded by source-localized electro/magnetoencephalography (EMEG), using computational models capturing semantic constraints derived from the prior context on each upcoming word. Our results provide insights into predictive operations subserved by different regions within a bi-hemispheric system, which over time generate, refine, and evaluate constraints on each word as it is heard.

Neural dynamics of semantic composition.

Human speech comprehension is remarkable for its immediacy and rapidity. The listener interprets an incrementally delivered auditory input, millisecond by millisecond as it is heard, in terms of complex multilevel representations of relevant linguistic and nonlinguistic knowledge. Central to this process are the neural computations involved in semantic combination, whereby the meanings of words are combined into more complex representations, as in the combination of a verb and its following direct object (DO) noun (e.g., "eat the apple"). These combinatorial processes form the backbone for incremental interpretation, enabling listeners to integrate the meaning of each word as it is heard into their dynamic interpretation of the current utterance. Focusing on the verb-DO noun relationship in simple spoken sentences, we applied multivariate pattern analysis and computational semantic modeling to source-localized electro/magnetoencephalographic data to map out the specific representational constraints that are constructed as each word is heard, and to determine how these constraints guide the interpretation of subsequent words in the utterance. Comparing context-independent semantic models of the DO noun with contextually constrained noun models reflecting the semantic properties of the preceding verb, we found that only the contextually constrained model showed a significant fit to the brain data. Pattern-based measures of directed connectivity across the left hemisphere language network revealed a continuous information flow among temporal, inferior frontal, and inferior parietal regions, underpinning the verb's modification of the DO noun's activated semantics. These results provide a plausible neural substrate for seamless real-time incremental interpretation on the observed millisecond time scales.

The Cortical Maps of Hierarchical Linguistic Structures during Speech Perception.

The hierarchical nature of language requires human brain to internally parse connected-speech and incrementally construct abstract linguistic structures. Recent research revealed multiple neural processing timescales underlying grammar-based configuration of linguistic hierarchies. However, little is known about where in the whole cerebral cortex such temporally scaled neural processes occur. This study used novel magnetoencephalography source imaging techniques combined with a unique language stimulation paradigm to segregate cortical maps synchronized to 3 levels of linguistic units (i.e., words, phrases, and sentences). Notably, distinct ensembles of cortical loci were identified to feature structures at different levels. The superior temporal gyrus was found to be involved in processing all 3 linguistic levels while distinct ensembles of other brain regions were recruited to encode each linguistic level. Neural activities in the right motor cortex only followed the rhythm of monosyllabic words which have clear acoustic boundaries, whereas the left anterior temporal lobe and the left inferior frontal gyrus were selectively recruited in processing phrases or sentences. Our results ground a multi-timescale hierarchical neural processing of speech in neuroanatomical reality with specific sets of cortices responsible for different levels of linguistic units.

Predictive Brain Mechanisms in Sound-to-Meaning Mapping during Speech Processing.

Spoken language comprehension relies not only on the identification of individual words, but also on the expectations arising from contextual information. A distributed frontotemporal network is known to facilitate the mapping of speech sounds onto their corresponding meanings. However, how prior expectations influence this efficient mapping at the neuroanatomical level, especially in terms of individual words, remains unclear. Using fMRI, we addressed this question in the framework of the dual-stream model by scanning native speakers of Mandarin Chinese, a language highly dependent on context. We found that, within the ventral pathway, the violated expectations elicited stronger activations in the left anterior superior temporal gyrus and the ventral inferior frontal gyrus (IFG) for the phonological-semantic prediction of spoken words. Functional connectivity analysis showed that expectations were mediated by both top-down modulation from the left ventral IFG to the anterior temporal regions and enhanced cross-stream integration through strengthened connections between different subregions of the left IFG. By further investigating the dynamic causality within the dual-stream model, we elucidated how the human brain accomplishes sound-to-meaning mapping for words in a predictive manner. SIGNIFICANCE STATEMENT: In daily communication via spoken language, one of the core processes is understanding the words being used. Effortless and efficient information exchange via speech relies not only on the identification of individual spoken words, but also on the contextual information giving rise to expected meanings. Despite the accumulating evidence for the bottom-up perception of auditory input, it is still not fully understood how the top-down modulation is achieved in the extensive frontotemporal cortical network. Here, we provide a comprehensive description of the neural substrates underlying sound-to-meaning mapping and demonstrate how the dual-stream model functions in the modulation of expectations, allowing for a better understanding of how the human brain accomplishes sound-to-meaning mapping in a predictive manner.

Functional Connectivity-Based Parcellation of the Thalamus: An Unsupervised Clustering Method and Its Validity Investigation.

Node definition or delineating how the brain is parcellated into individual functionally related regions is the first step to accurately map the human connectome. As a result, parcellation of the human brain has drawn considerable attention in the field of neuroscience. The thalamus is known as a relay in the human brain, with its nuclei sending fibers to the cortical and subcortical regions. Functional magnetic resonance imaging techniques offer a way to parcellate the thalamus in vivo based on its connectivity properties. However, the parcellations from previous studies show that both the number and the distribution of thalamic subdivisions vary with different cortical segmentation methods. In this study, we used an unsupervised clustering method that does not rely on a priori information of the cortical segmentation to parcellate the thalamus. Instead, this approach is based on the intrinsic resting-state functional connectivity profiles of the thalamus with the whole brain. A series of cluster solutions were obtained, and an optimal solution was determined. Furthermore, the validity of our parcellation was investigated through the following: (1) identifying specific resting-state connectivity patterns of thalamic parcels with different brain networks and (2) investigating the task activation and psychophysiological interactions of specific thalamic clusters during 8-Hz flashing checkerboard stimulation with simultaneous finger tapping. Together, the current study provides a reliable parcellation of the thalamus and enhances our understating of thalamic. Furthermore, the current study provides a framework for parcellation that could be potentially extended to other subcortical and cortical regions.

Cross-language differences in the brain network subserving intelligible speech.

How is language processed in the brain by native speakers of different languages? Is there one brain system for all languages or are different languages subserved by different brain systems? The first view emphasizes commonality, whereas the second emphasizes specificity. We investigated the cortical dynamics involved in processing two very diverse languages: a tonal language (Chinese) and a nontonal language (English). We used functional MRI and dynamic causal modeling analysis to compute and compare brain network models exhaustively with all possible connections among nodes of language regions in temporal and frontal cortex and found that the information flow from the posterior to anterior portions of the temporal cortex was commonly shared by Chinese and English speakers during speech comprehension, whereas the inferior frontal gyrus received neural signals from the left posterior portion of the temporal cortex in English speakers and from the bilateral anterior portion of the temporal cortex in Chinese speakers. Our results revealed that, although speech processing is largely carried out in the common left hemisphere classical language areas (Broca's and Wernicke's areas) and anterior temporal cortex, speech comprehension across different language groups depends on how these brain regions interact with each other. Moreover, the right anterior temporal cortex, which is crucial for tone processing, is equally important as its left homolog, the left anterior temporal cortex, in modulating the cortical dynamics in tone language comprehension. The current study pinpoints the importance of the bilateral anterior temporal cortex in language comprehension that is downplayed or even ignored by popular contemporary models of speech comprehension.